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1. Introduction 

Consider a nonlinear operator equation 
(1.1) F{x) =0, F:H-*H, 



in a real Hilbert space H. We suppose here that equation (1.1) is solvable (not necessar- 
ily uniquely), and the operator F is twice Frechet differentiable without such structural 
assumptions as monotonicity, invertibility of F' etc. 

In the theory of ill-posed problems various discrete and continuous methods based on a 
regularization are known. A principal point in the numerical implementation of regularized 
Newton's and Gauss-Newton's procedures is the inversion of the operators {F'{x) -\-£l) and 
{F'*{x)F'{x) -\- el) respectively Such an inversion for certain operators is a 

nontrivial task. Moreover, the inversion decreases the accuracy of computations. 

For well-posed equations (i.e., in the case when the Frechet derivative operator F'{x) 
is boundedly invertible in a ball, which contains one of the solutions) several approaches 
are taken in order to reduce the cost associated with the storage and inversion of F'{x) 
in Newton's scheme (or F'*{x)F' [x) in Gauss-Newton's scheme). Iterative techniques for 
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linear systems, like 'conjugate gradient', can be applied to compute an approximation to the 
Newton step s„ := Xn+i — Xn, yielding an inexact, or truncated, Newton's method. Such a 
method does not explicitly require the operator F'{x). Instead, it requires only applications 
of i^'(a;) to certain elements. See g, or (||], p.l36). 

Another approach is to replace the exact Frechet derivative by an approximation obtained 
from current and previous values of F{x). In this case, only F{xn+i) and F{xn) are required. 
This generates the secant method (0, p. 201). Perhaps the most popular secant method is 
the BFGS method, which was discovered independently by Broyden, Fletcher, Goldfarb, 
and Shanno in 1970. To implement the BFGS, given an approximation J„ to F'{x„), one 
first computes 

Sn ■■= Xn+1 - Xn and y„ := F{Xn+l) - F{Xn). 
One then gets an approximation to F'{xn+i): 



If J„ is symmetric positive definite and y^Sn > 0, then J„+i will also be symmetric positive 
definite. In the finite dimensional well-posed case, under standard assumptions, the BFGS 
method is guaranteed to be superlinearly convergent. 

BFGS has a limited memory variant (|^, p. 224), which requires no explicit matrix storage 
for the approximate derivative. It is based on the recursion for the inverse 



In ^ the following continuous Newton-type algorithm for solving nonlinear well-posed op- 
erator equations is investigated 

x{t) = -J{t)F{x{t)), x{0) = xo e H, J(0) G L{H), 

j{t) = -p^[F'*{x{t))F'{x{t))J{t) + J{t)F'{x{t))F'*{x{t))] + 2p^F'*{x{t)). 

The theorem establishing convergence with the exponential rate is proved for the above 
procedure. 

In many important applications the Frechet derivative operator is not boundedly invert- 
ible: this is an ill-posed case. In order to deal with this difficulty, we propose a novel 
continuous algorithm with simultaneous updates of {F'* {x{t))F' {x{t)) +e{t)I)~^ and prove 
a convergence theorem. The paper is organized as follows. In section 2 we introduce the 
algorithm and also state two Lemmas. In section 3 the main convergence result is estab- 
lished. An attractive feature of this result (formulas ( ^.3| )-(^!^) in section 2) is the absence 
of the assumptions about the location of the spectrum of the operator F'(x). For example, 
in 1^ the operator F'(x) was assumed nonnegative, which is a strong assumption about 
location of its spectrum. One may generalize the methods and results of Q to the case of 
the operators for which 5RF'(x) > 0, which still imposes a restriction on the location of the 
spectrum of F'{x). The absence of such assumptions in our paper is made possible by the 



conditions 5) and 6) of Theorem 3.1 in section 3 



2. The Initial Value Problem and Some Auxiliary Results 



Consider continuously regularized Gauss-Newton's procedure for solving equation (1.1) 
(2.1) x{t) = - [F'*{x{t))F'{x{t)) + e{t)l] [F'* {xit))Fixit)) + £(i)(a;(t) - xo)] , 
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where £{t) > 0, xq £ H, and / is an identity operator. The reader may consult ||^ and 
for a convergence analysis of (2.1). Problem is equivalent to the following system: 



x{t) - -Q{t)[F'*ix{t))F{x{t)) + e{t){x{t) ~ xo)], 



(2.2) 



[F'*ixit))F'ixit)) + e{t)I]Q{t) - / - 0, Q{t) e L{H). 



Solve equation (2.2) by using continuous simple iteration scheme. We arrive at the initial 
value problem 



(2.3) 



x{t) = -Bit)[F'*ixit))Fixit)) + eit)ixit) - xo)], 



(2.4) 



m = - [{F'*{xit))F'{x{t)) + e{t)r)B{t) I] , 
x{0)^xoeH, B{0)<eL{H), 0<e{t)^0 as t ^ +oo. 



The following lemma about the inversion of a nonlinear differential inequality was first 
stated and proved in 

Lemma 2.1. Let "f{t), a{t), l3{t) G C[0, +oo). // there exists a positive function fi(t) G 
C"'^[0, +cxd) such that 



(2.) o,.,o.f ,„o.^(„0-f 

then a nonnegative solution to the following inequality 

(2.6) v{t) < -i{t)v{t) + a{t)v^{t) + (3{t) 

satisfies the estimate: 



(2.7) 



< v{t) < 



1 



Now, we formulate the second lemma, which is an operator-theoretical version of the 
well-known Gronwall inequality: 



Lemma 2.2. Let 

(2.8) 



^ + Ait)Vit) = Git), ViO) = Vo, 



where A{t), G{t), V{t) e L{H), and H is a real Hilbert space. If there exists ^(t) > such 
that 



(2.9) 

then 

(2.10) 



{A{t)h,h)>j{t)\\h\\^ yheH, 



-J l(p)dp 

\\vm<e « 



J l(p)dp 

||G(s)||eo ds + \\Vm\ 
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Proof. Take any h ^ H. Since _ff is a real Hilbert space one has: 

= -{A{t)V{t)h,V{t)h) + {G{t)h,V{t)h) 
(2.11) <-7W||F(^)/^|p + ||G(^)|| \\h\\ \\Vit)h\\ 

Denote v{t) := \ \V{t)h\\. Inequality ( ^.11 ) implies 



(2.12) vi! < -7(i)w^ + l|G(OI| \\h\\v. 

Divide this inequality by the nonnegative v and get a linear first-order differential inequality 



from which one can derive the desired inequality (2.10). Lemma 2.3 is proved. □ 



3. The Convergence Theorem 

Theorem 3.1. Let H be a real Hilbert space, F : H ^ H . 

1) Assume that < £{t) G C"'^[0, +cxd) converges to zero monotonically and 

(3.1) \eit)\<b£^{t) for any te[0,+oo), 
where b > is a constant. 

2) Suppose problem ( jj.j[ ) is solvable (not necessarily uniquely) and x is its solution. 

3) There exists a positive number R such that F is twice Frechet differentiate in a closed 
ball U{x,Re{0)) { X : X £ H, \\x-x\\ < Re{0)} ana, 

\\F'{x)\\ < Ni, ||i^"(a;)|| < A^2 for all x e U{x, Rs{0)). 

4 ) Let the following inequality be satisfied 

(3.2) fc + &£(0)<l, 

where k 2N1N2R + b + e{0)\\B{0)\ \ + \\L - B{0)[F'*{x)F'{x) + e{0)L]\\. 

5) For an initial approximation point Xf) the source-type condition holds 

(3.3) xo G U{x, Re{0)) n[x + Ran{F'*{x)F'{x))] , 

where Ran{F'*{x)F'{x)) is the range of the linear operator F'*(x)F'(x), and for some w, 
.such that X ~ xq ~ F'*{x)F'(x)w, one has: 

(3 4) 1 ^ 3jViiV2(l + £(0)||i3(0)||) ^^.^f l-k-be{0) e{0) 



R- l-fc-6e(0) i2(/c + 2 + e(0)||B(0)||)||«7||' ||a;(0) -i| 

Then 



the solution {x{t), B(t)) to problem (2.S)-(2.4) exists for all t G [0, +cxd) and 



(3.5) \\xit) ~ x\\ < Rsit). 



Remark 3.2 There are many e{t) satisfying condition 1) of Theorem 3.1. For example, one 
may take e{t) = co(ci + i)^"^, where cq and Ci are positive constants and a G (0, 1]. 



Remark 3.3 Note that assumption (3.3) is not algorithmically verifiable. However, practi- 
tioners may try different xq and choose the one for which the algorithm works better, that 
is, convergence is more rapid and the algorithm is more stable. 

If F'*{x)F'{x) is compact and the null space N{F'*{x)F'{x)) = {0}, then the range 
Ran{F'* {x)F' (x j) is dense in H, so in any neighborhood of x there are points xq for which 
(O) holds. On the other hand, since F'*{x)F'{x) is compact, the set R{F'*{x)F'{x)) is 
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not closed. Thus, in the same neighborhood there are also points xq for which (3.3) fails to 
hold. In general, in order to get a convergence theorem in an ill-posed case one needs some 
additional assumptions on the Frechet derivative of the operator F, for example condition 
(^), or some other condition of this type ( see e.g., |8|, condition (2.11)). 
Remark 3.4 By the assumption 4) of Theorem |3.l| the first inequality in (3.4) is equivalent 
to the following one: 

(3.6) 3A^iA^2i?(l + e(0)||B(0)||) > 1 - 2N1N2R - b ~ e(0)||B(0)|| - ||A(0)|| - te(0), 
where ||A(0)|| := \\I - B{0)[F'*{x)F'{x) + e{0)l]\\. Inequality (|!|) yields 

l-6-£(0)||i?(0)||-||A(0)||-M0) 



(3.7) 
If one takes 
(3.8) 



R > 



R := 



(5 + 3£(0)||B(0)||)iVi7V2 
l-6-£(0)||S(0)||-||A(0)||-te(0) 



{5 + 3e{0)\\B{0)\\)NiN2 



then condition (3.2) is equivalent to 
(3.9) 6+||A(0)||+5e(0)[l + ||B 
For inequality 



S(0)||[e(0)||i3(0)|| + ||A(0)||+e(0)]<l. 



to hold one has to have b, 
and I 



|A(0)1| and e(0)||B(0)|| sufficiently small. 
|xo —x\ \ are sufficiently small. Such a priori 



The second inequality in (3.4) holds if ||w 
assumptions are typical for the methods of solving nonlinear problems, in particular, for the 
Newton-type methods. 



Remark 3.5 In practice, algorithm (2^)-(2^) proved to be efficient. In order to test ( p.3|) - 
( ^.4| ) numerically, we considered a two-dimensional inverse gravimetry problem (see , |]lO| ) 
and the Feigenbaum equation (see equation (5.1)). The goal of th e experiments was to 
investigate the choice of t he regulariz atio n function e{t) in ( |2.3| )-(2.4) and to compare two 
continuous methods: ( ^.3[ )-(p^) and {2A). Analyzing the results of practical computations 
we arrived at the following conclusions: 



1. 



Continuously regularized Gauss-Newton's scheme (2.1) tends to do well initially, until 
the value of a regularization function s{t) is not very small. Continuous procedure 
(2.3)-(2.4) is more stable, and allows one to get the approximate solution with better 
accuracy. 

2. The best numerical results for ( ^.3[ )-(p^) are obtained with e{t) = e(0)(l + t)^^. An 
approximate range of possible values for e(0) appears to be from 0.001 to 0.1; for larger 
values the accuracy is lower, and for smaller values the process does not converge. 

3. An important task that should be studied next is the stability of the solution towards 
the noise in the data, and the choice of an optimal regularization parameter (the 
stopping time in our case) such that the method converges to the solution of (1.1) 
when the noise level tends to zero. Also one has to a naly ze un der what assumptions on 
F and xq the convergence of the continuous process (2.3)-( p^ implies the convergence 
of the corresponding discrete process, obtained by the Euler or the Runge-Kutta finite 
difference methods. 

Proof of Theorem |3.l|. T he main part of the proof (Steps 1-3) is to show that if 
{x(t),B{t)) is a solution to (^.3|)-(2.4), then \\x{t) — x\\ < Re(t). Assume the converse. By 
(|^) one has the inequality ||a;(0) — a;|| < Re{0). Therefore there exists a r G [0, +00), such 
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that 

(3.10) \\x{t) - x\\ < Re{t) yte[0,T) and \\x{t) - x\\ ^ Re{T). 

Step 1. In this step we derive some estimates ( see ( 3.11 )), ( |3.13| ) and ( 3.14 ) below) for 

||i?(t)||, \\I-B{t)[F'*ix)F'ix)+eit)l]\\ and \\B{t)F'* {x)F' {x)\\- 
We assume below that t G [0, t]. Applying Lemma |2.2| with 

Vit) := B{t), A{t) F'* {x{t))F' {x{t)) + e{t)I, G{t) := I, 

one gets 



mt)\\<e ° 



r Je{p)dp 

eo ds+\\BiO)\\ 



- I s(p)dp 

< e ° 



1 

W) 



J e(p)dp 

e[s)e° ds + \\B{G)\\ 



1 

W) 



(3.11) 

Introduce the notation 

(3.12) 

Then 



- j e(p)dp 

1 - e ° 



— f e(p)dp 1 

+ ||i3(0)||e < +115(0)11. 



A{t) := / - B{t) [F'*{x)F'{x) + e{t)l] . 



A(i) = ~B{t)[F'*{x)F'{x)+e{t)l] - B{t)e{t) 
= [F'*{x{t))F'{x{t))B{t) -1 + e{t)B{t)] [F'*{x)F'{x) + e{t)l] - B{t)e{t) 
= [F'*{x{t))F\x{t)) + e{t)l] {I - A{t)) - [F'*{x)F'{x) + e{t)l] - B{t)e{t) 
= -[F'*{x{t))F'{x{t)) + e{t)l]k{t) + F'*{x{t))F\x{t)) - F'*{x)F'{x) - B{t)e{t). 
Use Lemma |2.2| once again, with 

Vit) A(i), Ait) F'*(x(t))^^'(a;(t)) + e(t)/, 
Git) := F'*ixit))F'ixit)) - F'*ix)F'ix) - B(t)£(i). 

From the estimate 

\\F'*ih)F'ih)-F'*iy)F'iy)\\ = \\F'* ih)[F' ih) - F' iy)] + [F'* ih) - F'* iy)]F' iy)\\ 
< 2iViiV2||^-y|| for any /i, y e [/(x, i?e(0)), 



and from (3.11), (3.10), ( ^.l| ) and condition 3) of Theorem 3.1 one obtains 
t 

' ' ' 'mo)\\ 



- 1 ^(p'l^p 

||A(t)||<e 



f e(p)dp 

eo ds+ ||A(0)|| ^ < e 



2NiN2\\xis) ~ x\\ + 

t 

- I e(p)dp 



sis) 



1 \ - S <^{p)dp 

e " 



sis) 



\£[S 



t s t 

f E{p)dp 

i2NiN2R + b) I e(s)eo ds ^ 



\eis)\ds + \\m\\ 



i2NiN2R + b) 



- I <^{p)dp 

1 - e « 



\\BiO)\\ 



eiO) 



l|S(0)||-^ 

£(s) 



(e(0) -£(<))+ 



-J e(p)dp 

(3.13) +||A(0)||e <27ViiV2i? + 6 + e(0)||S(0)|| + ||A(0)|| :=fc 



Now denote D{t) B{t)F'*{x)F'{x). Then by ||A(t)|| = \\I - D{t) - B{t)eit)\\ < k, 

and therefore by 

(3.14) \\Dit)\\<k + 2 + em\BiO)\\. 

Step 2. In this step we derive the differential incquahty (see ( 3.20| ) below). 
Since H is a real Hilbert space one has: 

~\Kt) - i|P = ix,x{t) -x) = -{Bit)[F'*ix{t))F{x{t))+e{t){x{t) - xo)],x{t) - x). 

One gets 
(3.15) 
where 

(3.16) 



F{x{t)) = F\x){x{t) -x)+ K{x{t),x{t) - X, x{t) - x), 



\\K{h,y,z)\\ < 



N2 



z\\ for any h £ U{x, Re{Oj), y,zeH. 



Thus, (|1|), ( |3l6| ) and ( |3ll| ) yield 

■j^Mt)-x\\^<-{Bit 
iVi7V2(l + £(0)||S(0)||) 



^^lk(t)-i|P < -{Bit)F'^x{t))F\x){x{t)-x),x{t)--x)~e{t)iB{t){xit)~xo),x{t)-x) 



2e{t) 



\x{t) - x|p = -{B{t){F'{x{t)) - F'{x))*F'{x){x{t) ~ x), x{t) ~~ x) 



-iB{t)[F'*{x)F'{x) + e{t)l] {x{t) - x),x{t) - x) - e{t){B{t){x ~ xo),x{t) - x) 



jVijV2(l + £(0)11^(0)11) 
2e{t) 



\\x{t)-x\ 



From ( |3.12 ) one gets 

-{B{t) [F'*{x)F'{x) + e{t)l] {x{t) - x),x{t) ^ x) ^ -\\x{t) - xW" 

(3.17) +{K{t){x{t)-x),x{t)-x). 
By estimate (3.11) 



(3.18) 



-{B{t){F'{x{t)) - F'{x))*F'{x){x{t) - x),x{t) - x) 
7Vi7V2(l + e(0)||B(0)||), 



< 



e{t) 



\x{t)-x\\' 



Condition 5) of Theorem |3.l| implies: 

(3.19) ~e{t){B{t){x - xo),x{t) ~ x) ^ -e{t){D{t)w,x{t) -x). 

From (|t3), (|3T|), ( FTsI) , (|3t|), (|t|) one concludes 



1 d 



\x{t)-xf< 



37ViAr2(l + £(0)||B(0)||), 



a;(t)-£|p-(l-fc)||x(i)-x|| = 



2 - 2e(i) 

+£(i)(A: + 2 + £(0)11^(0)11) \\x{t)-x\\. 
Let w(t) := \ \x{t) — x\\. Then we arrive at the inequality 



Step 3. In this step we choose /i(t) — > cx) as i ^ cx), such that assumptions (2.5) of 
Lemma 2.1 are satisfied for inequality (3.20), so that the conclusion ||a;(i) — x|| < ^^^j 
holds. 
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Choose iJ,{t) e C^[0,+oo) to satisfy conditions (2.5) of Lemma 2.1 with 

3iViiV2(l + £(0)||S(0)||) 



(3.21) 



lit):^l~k, ait).-- ^^^^^ 

Pit) :=£(t)(fc + 2 + £(0)||i?(0)||)||«;||. 



(3.22) 

Take ii{t) — where A > is a constant. Then ( |2.5| ) can be rewritten as follows: 



(3.23) 
(3.24) 
(3.25) 



3jVijV2(l + £(0)11^(0)11) ^ 
2e{t) 

e{t){k + 2 + em\Bm\)M\ < 
X 



2e{t) 
2A 

||x(o)-i||<i. 



i-fc 



1 - k 



\m\ 

e{t) 

\m\ 

e{t) 



Inequality ( 3.23| ) is equivalent to the following one 
(3.26) ^^_3A^iA^2(l + £(0)115(0)11) 



e(t) 



37Vi7V2(l+e(0)||S(0)||) 



If one takes 

•■• 1-fc-Mo) ' 

then ( |3.26| ) follows from (3.27) and from conditions 1) and 4) of Theorem B.l. Assumption 
( p^ ) of Theorem 3.1 implies that 

(3 28) 3jVijV2(l + £(0)||i3(0)||) ^ l-k~ be{0) 



l-k-be{0) ^ 2(A: + 2 + £(0)||B(0)||)||w|r 

For A defined by (|3.27|) inequality ( ^.28| ) can be written as 

1 - fc - 6£(0) 

^ • ^ 2(fc + 2 + £(0)||i3(0)||)|Hr 

From (|3.29|) one obtains ( ^.24[ ). 

Finally, from ( |3.4|) one concludes that 



(3.30) 



3iViiV2(l + £(0)||B(0)||) 



< 



£(0) 



l-A:-5£(0) ^ ||x(0)-x||' 

which is equivalent to ( 3.25| ) for A defined by ( 3.27 ). 
Hence, by Lemma 2.1, one obtains: 

ll^(i)_i||<£|). 



By (|3.27| ) and (|3. 

\\x{t^^x\ 



1 - fc - 6£(0) „ n „ r„ n 

< -, , , ,,,, < R for all ieO,T, 

e{t) 37Vi7V2(l+£(0)||B(0)||) - ^ ' ^' 

which contradicts ( 3.10| ). Thus \ \x(t) — x\ \ < Rs{t) whenever x{t) is defined. 
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Step 4. Now let us show that there exists the unique solution to ( p.3[ )-(p.4|) on [0, +cxo). 
The initial value problem (2_^)-(2^) is equivalent to the integral equations: 

t 



(3.31) 



x{t)^XQ+ / B{s)^{s,x{s))ds, xq <E H, 



(3.32) 
where 



B{t) = B(0) + j [ip{s, x(s))B{s) + I]ds, B{0) e L{H), 



$(s, x) := -[F'*{x)F{x) + e{s){x - xq)], 
(p{s,x) ~[F'*{x)F'{x)+e{s)I]. 

We have already proved that if for some positive number T the solution {x{t), B{t)) to 
problem (2.3)-(2.4) exists on [0,r], then x{t) and B{t) are bounded in the norm on [0, T]. 
Therefore, since the integrands in ( ^.31 ) and ( 3.32| ) are Lipschitz-continuous, the standard 
argument yields existence and uniqueness of the global solution to system ( ^.31 )-(3.32). This 
concludes Step 4. 

For convenience of the reader we give a complete proof of Step 4. Let us fix an arbitrary 
large positive number T and use the successive approximation method to solve equations 
(^)-(|33^) on [0,r]: 

t 

(3.33) Xn+iit) = xo+ Bn{s)^{s,Xn{s))ds, Xo E H, 



(3.34) B^+^{t)^B{Q)+ j[ip{s,x^{s))B^{s) + I]ds, B{0) e L{H). 



Since F'{x) and F"{x) are assumed to be bounded in U{x, Re{0)), and e{t) is positive on 
[0,T], for every t G [0,T] the function ^{s,x) has bounded Frechet derivative with respect 
to X in U{x, Re{0)). So one has: 

\\B2Hs,X2) - -Bi$(s,xi)|| = 11(52 - Si)$(s,X2) +Bi($(s,a;2) - $(s,xi))|| 

< A(r)(||i?2-i?i|| + ||a;2-xi||) 
for aU t e [0,r] and xi,X2 E U{x,Re{0)). Similarly, 

\\(p{s,X2)B2 - ip{s,Xi)Bi\\ = \\{(p{s,X2) - (p{s,Xi))B2 + ip{s,xi){B2 - Bi)\\ 

<K{T){\\B2-Bi\\ + \\x2-xi\\). 
Define the norm in a Banach space H x L{H) as follows: 

\\{x,B)\\:=\\x\\ + \\B\\. 
Thus, if Wi{t) := {xi{t), Bi{t)), i = 1,2, one gets the estimate 



\\wn+i{t) - Wnit)\\ < / ||B„(s)$(s,x„(s)) -B„_i(s)$(s,a;„_i(s))||ds 
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\\ip{s,Xn{s))Bn{s) - Lp{s, Xn^lis))Bn_i{s)\\ ds 



< J{A{T) + K{T))\\w,,is)-w,,^,is)\\ds <...< i^Sll±^iD)Dl const 



valid on the maximal subinterval [0,T,„aa;] ■— {t, t e [0, T] and x{t) G U{x, Re{Oj)}. There- 
fore iterative process ( ^.33D-(^34 ) converges uniformly and determines the unique solution 



{x{t),B{t)) of problem (^.3"ll)-(^.32|) on [Q,Tmax\- If Tmax < T, it follows from the maxi- 



mality of the subinterval [0,Tmax] that x{Tmax) is a boundary point of U{x, Re{0)). But, 
since e{t) is monotone, this contradicts the fact that the inequality \ \x{t) — x\ \ < Re{t) holds 



whenever x{t) is defined. Hence the unique solution {x{t),B{t)) to (|2.3|)-(2.4) exists on 
[0,T] for any T > 0. Consequently, it exists on [0, +00) and ( |3.5| ) is satisfied Vi € [0, +00). 
Theorem ^T] is proved. □ 
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